Search Results: "David Bremner"

25 October 2008

David Bremner: Tunnel versus MUX: a race to the git

Third in a series (git-sync-experiments, git-sync-experiments2) of completely unscientific experiments to try and figure out the best way to sync many git repos. If you want to make many ssh connections to a given host, then the first thing you need to do is turn on multiplexing. See the ControlPath and ControlMaster options in ssh config Presuming that is not fast enough, then one option is to make many parallel connections (see e.g. git-sync-experiments2). But this won't scale very far. In this week I consider the possibilities of running a tunneled socket to a remote git-daemon
ssh  -L 9418:localhost:9418 git-host.domain.tld git-daemon --export-all
Of course from a security point of view this is awful, but I did it anyway, at least temporarily. Running my "usual" test of git pull in 15 up-to-date repos, I get 3.7s versus about 5s with the multiplexing. So, 20% improvement, probably not worth the trouble. In both cases I just run a shell script like
  cd repo1 && git pull && cd ..
  cd repo2 && git pull && cd ..
  cd repo3 && git pull && cd ..
  cd repo4 && git pull && cd ..
  cd repo5 && git pull && cd ..

8 October 2008

David Bremner: The mailbox plugin for ikiwiki

In a recent blog post, Kai complained about various existing tools for marking up email in HTML. He also asked for pointers to other tools. Since he didn't specify good tools :-), I took the opportunity to promote my work in progress plugin for ikiwiki to do that very thing. When asked about demo sites, I realized that my blog doesn't actually use threaded comments, yet, so made a poor demo. Follow the link and you will find one of the mailboxes from the distribution, mostly a few posts from one of the debian lists. The basic idea is to use the Email::Thread perl module to get a forest of thread trees, and then walk those generating output. I think it would be fairly easy to make a some kind of mutt-like index using the essentially same tree walking code. Not that I'm volunteering immediately mind you, I have to replys to comments on my blog working (which is the main place I use this plugin right now).

26 September 2008

David Bremner: Can I haz a distributed news reader?

RSS readers are better than obsessively checking 18 or so web sites myself, but they seem to share one very annoying feature. They assume I read news/rss on only one machine, and I have to manually mark off which articles I have already read on a different machine. Similarly, nntp readers (that I know about) have only a local idea of what is read and not read. For me, this makes reading high volume lists via gmane almost unbearable. Am I the only one that wastes time on more than one computer?

20 September 2008

David Bremner: managing many git repos

I have been thinking about ways to speed multiple remote git on the same hosts. My starting point is mr, which does the job, but is a bit slow. I am thinking about giving up some generality for some speed. In particular it seems like it ought to be possible to optimize for the two following use cases: For my needs, mr is almost fast enough, but I can see it getting annoying as I add repos (I currently have 11, and mr update takes about 5 seconds; I am already running ssh multiplexing). I am also thinking about the needs of the Debian Perl Modules Team, which would have over 900 git repos if the current setup was converted to one git repo per module. My first attempt, using perl module Net::SSH::Expect to keep an ssh channel open can be scientifically classified as "utter fail", since Net::SSH::Expect takes about 1 second to round trip "/bin/true". Initial experiments using IPC::PerlSSH are more promising. The following script grabs the head commit in 11 repos in about 0.5 seconds. Of course, it still doesn't do anything useful, but I thought I would toss this out there in case there already exists a solution to this problem I don't know about.
 
#!/usr/bin/perl
use IPC::PerlSSH;
use Getopt::Std;
use File::Slurp;
my %config; 
eval( "\%config=(".read_file(shift(@ARGV)).")");
die "reading configuration failed: $@" if $@;
my $ips= IPC::PerlSSH->new(Host=>$config host );
$ips->eval("use Git");
$ips->store( "ls_remote", q my $repo=shift;
                       return Git::command_oneline('ls-remote',$repo,'HEAD');
                            );
foreach $repo (@ $config repos ) 
    print $ips->call("ls_remote",$repo);
 
P.S. If you google for "mr joey hess", you will find a Kiss tribute band called Mr. Speed, started by Joe Hess" P.P.S. Hello planet debian!

David Bremner: managing many git repos II

In a previous post I complained that mr was too slow. madduck pointed me to the "-j" flag, which runs updates in parallel. With -j 5, my 11 repos update in 1.2s, so this is probably good enough to put this project on the back burner until I get annoyed again. I have the feeling that the "right solution" (TM) involves running either git-daemon or something like it on the remote host. The concept would be to set up a pair of file descriptors connected via ssh to the remote git-daemon, and have your local git commands talk to that pair of file descriptors instead of a socket. Alas, that looks like a bit of work to do, if it is even possible.

Next.

Previous.